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® A video processing system. 



@ A video processing system comprising a pro- 
cessing unit 12 a VDU 16 and a look-up table 17. A 
user defined profile is displayable on the VDU 16 
and can be adjusted interactively by way of a stylus 
14 and touch tablet 15. Data representing the profile 
is stored in the look-up table. An input video clip is 
stored in digital form in a disc store 10 and output 
video clip frames are generated In accordance with 
the user defined profile data. The system can be 
used for example to stretch or compress a video clip 
in time. 
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A VIDEO PROCESSING SYSTEM 



FIELD OF THE INVENTION 

The invention relates to a video processing 
system and in particular relates to a system in 
which an input sequence of video frames is pro- 
cessed to produce an output sequence. 



BACKGROUND OF THE INVENTION 

Editing techniques are known in which the nor- 
mal sequence of video frames from a video clip is 
altered to produce a re-timed sequence. For exam- 
ple a clip having a duration of only a few seconds 
may be repeated or individual frames of a clip may 
be removed to reduce slightly the duration of the 
clip. These techniques are often employed to syn- 
chronise video to an audio soundtrack and in more 
sophisticated applications for example to synch- 
ronise the movements of a mouth (possibly that of 
an animal) to a recorded soundtrack, of say, 
speech. The legs of an animal may also be synch- 
ronised in this way so that a single leg movement 
may be repeated many times to create, say, the 
Illusion of a cow 'dancing' in time to recorded 
music. 

A machine which has made these effects, and 
many more, available to commercial video produc- 
ers is manufactured by the present Applicant and 
sold under the trademark "HARRY". In this ma- 
chine, frames of video are digitally encoded and 
stored on magnetic discs, as disclosed in US Pat- 
ent No. 4,688,106 assigned to the present ap- 
plicant. A problem with known techniques for 
stretching video clips is that each stretching (or 
compressing) process has been limited to very 
simple functions, such as stretch by 20%, or re- 
move 3 frames etc. Thus, a complicated movement 
lasting only a few seconds may have to be built up 
from many small clips having different functions 
applied thereto. 



OBJECTS AND STATEMENTS OF THE INVEN- 
TION 

The Invention aims to provide an improved 
system for stretching and compressing video clips. 

According to the present invention, there is 
provided a video processing system, comprising a 
video storage device for storing an input sequence 
of video frames; defining means for defining an 
output sequence of video frames in relation to said 
input sequence frames; and generating means for 
generating an output sequence of video frames 



from said input video frames in accordance with 
the defined relationship. 

The above and further features of the invention 
are set forth with particularity in the appended 
5 claims and together with advantages thereof will 
become clearer from consideration of the following 
detailed description of an exemplary embodiment 
of the invention given with reference to the accom- 
panying drawings. 

10 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 shows a video processing system 
75 including a graphical device for displaying func- 
tional profiles; 

Rgure 2 shows examples of graphical pro- 
files of the type displayed on the device shown in 
Rgure 1 ; and 

20 Figure 3 shows an enlarged portion of a 

profile shown in Figure 2. 

DETAILED DESCRIPTION OF AN EMBODIMENT 
OF THE INVENTION 

25 

A video processing system is shown in Rgure 
1 in which an input sequence of video frames, 
referred to herein as an input clip, is supplied to a 
parallel magnetic disc store 10 from a video tape 
30 recorder VTR 11. The video data is stored on disc 
as digitally encoded pixels, each having compo- 
nents representing luminance and two colour dif- 
ference values. A system for storing video data on 
magnetic disc, allowing data transfer at video rate, 
35 is disclosed in United States Patent No. 4.688,016 
assigned to the present Applicant. Alternatively, the 
data may be stored in a solid-state memory device. 
Preferably, the video data is in the abovementioned 
component form but a D2 composite machine, or 
40 an analogue machine, may instead be used to- 
gether with a suitable converting device for con- 
verting the data into component fonn. 

Data from the disc store 10 is read by a 
processing unit 12. The processing unit 12 is ar- 
45 ranged to convert the video data from its compo- 
nent form into analogue signals which are applied 
to a monitor 13 for display of the video image on 
the monitor 13. The monitor 13 is arranged also to 
display symbols representing user selectable con- 
so trol functions in the form of a menu of options from 
which a function can be selected. A stylus 14 and a 
touch tablet 15 are provided in order to facilitate 
control of the system. Positioning and movement of 
the stylus 14 on the touch tablet 15 causes the 
touch tablet 15 to generate signals representative 
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of the position of the stylus 14 and these signals 
are interpreted by the processing unit 12 to cause 
a cursor (not shown) to be displayed at a cor- 
responding position on the monitor 13. The stylus 
13 is pressure sensitive and values representing 
the pressure applied by the stylus to the touch 
tablet 14 are also delivered to the processing unit 
12. In order to select an option from the menu, the 
cursor is moved over the representation in the 
menu of the desired option by way of movement of 
tiie stylus on the touch tablet and the pressure 
applied by the stylus is increased by pressing 
down hard. The processing unit 12 responds to this . 
by reconfiguring the system to perform the se- 
lected function. 

A function which can be selected, from the 
menu displayed on the monitor 13, is "profile 
stretch" in which an input video clip is stretched, 
compressed or reversed in response to a profile 
defined by the user of the system. \/Vhen "profile 
stretch." is selected the processing unit 13 causes 
a visual display unit. VDU 16 to display a profile 
graph defined by mutually perpendicular x and y 
axes, in which the vertical y axis represents the 
frames of a stored input video clip and the horizon- 
tal X axis represents the frames of a video clip to 
be generated and output for display. 

The profile graph displayed on the VDU 16 is 
shown In detail in Figure 2 of the accompanying 
drawings. Figure 2 also shows three exemplary 
user defined profiles 21. 22 and 23. A profile defin- 
ing the relationship between input video frames 
and output video frames can be defined by the 
user by way of operation of the stylus 14 upon the 
touch tablet 15, with movement of the stylus being 
interpreted by the processor 12 as corresponding 
changes In the profile. The definition of the profile 
graph on the VDU 16 is preferably 256 x 256 
points and therefore a profile will consist of lines 
connecting points on a 256 x 256 grid. Other grid 
definitions may be used but the 256 grid Is suitable 
for most applications. As a profile line Is drawn the 
position of the stylus along the y axis is stored for 
each of the 256 positions along the x axis - the y 
scale also being divided into 256 positions. 

Of the three profiles shown In Figure 2, the 
profile 21 (y = x) produces no effect because the 
number of each output video clip frame corre- 
sponds to the number of each input video clip 
frame. The profile 21 is therefore an Identify func- 
tion. 

The profile 22 comprises a first portion 23 and 
a second portion 24 connected at a knee point 25. 
The first portion 23 causes a first portion of an 
input video clip comprising a number of frames to 
be stretched over a larger number of output 
frames, so that on playback the first portion frames 
of the clip are displayed on the monitor at a slower 



rate. After the knee point 25, the remaining frames 
of the input video clip are compressed in time for 
display in the remainder of the output clip. The 
remaining frames are therefore seen at increased 
5 speed. 

The profile 23 first causes a portion of the clip 
to be output at a faster rate, it then causes the 
output clip to be the reverse of the input clip for a 
while, it next causes a portion of the output clip to 
10 be an even faster version of a corresponding por- 
tion of the input clip, for a short period a portion of 
the output clip is again the reverse of a con'e- 
sponding input clip portion, and finally the last few 
frames at the output clip are a speeded up version 
75 of the corresponding last few input clip frames. 
Each of the profiles 21 to 23 shown produces an 
output which starts at the first frame of the input 
clip and ends at the last frame of the input clip. 
However a profile does not necessarily have to 
20 conform to this constraint and a profile may be 
drawn anywhere within the area bounded by the 
axes, subject to the restriction that the Input frames 
(y) must be defined as a continuous function of 
output frames (x) with only one value of y for each 
25 value of X. 

An input clip may consist of any number of input 
video frames and similarly an output clip may be 
defined as any number of output video frames; the 
number of input frames being defined when the 
30 clip is selected and the number of output frames 
being selected by the operator. The scale of the 
profile graph is for example 256 by 256 points and 
the profile is stored by programming a look-up 
table 17. The look-up table 17 consists of a random 
35 access memory device having 256 memory loca- 
tions one for each x axis value and each location 
being arranged to store an eight bit code repre- 
senting a corresponding y axis value as defined by 
the profile. Memory locations are addressed by 
40 supplying an 8 bit address code corresponding to 
an X axis value. That Is to say, an eight bit code 
representing a value on the x axis is applied to the 
address bus of the memory device to produce at 
the memory output an eigtit bit code representing a 
45 value on the y axis. 

A profile is defined by use of the stylus 14 and 
the touch tablet 15. Points can be defined at any 
desired position on the graph and when a point is 
selected, the x value at that point is used to ad- 
50 dress a location in the look-up table 17 and the 
corresponding y value is stored as an eight bit 
code at that location. When a point is defined by 
the user, the profile between the defined point and 
the previously define point is displayed by drawing 
55 a straight line (i-e. by linear interpolation) between 
the two points. With alt desired points in the profile 
selected, the profile may be smoothed between 
points by applying a smoothing function, for exam- 
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pie least squared fit, to the selected points. 
Predefined profiles such as a sinewave portion or 
other commonly used functions can also be stored 
in memory and selected by the user when re- 
quired, instead of having to define a profile each s 
time the "profile stretch" is to be used. Once the 
profile has been defined, an input clip is Identified 
from the video data in the disc store 10 using the 
stylus and touch tablet interactively with the moni- 
tor 13. This interactive identification consists of the io 
user Identifying the desired start frame and end 
frame of the input clip from the available video in 
the disc store 10, thereby defining the number of 
frames in the input clip. The number of output 
frames is then selected and a check is made to 75 
ensure that sufficient space exists in the disc 10 to 
accept these new frames as they are generated. 

In the case of the three profiles 21. 22, 23 in 
Figure 2, the first frame of the output clip is iden- 
tical to the first frame of the input clip, because 20 
each of the profiles 21, 22. 23 starts at the origin of 
the profile graph. However, with the exception of 
the profile 22, the second frame of the output clip 
will not be the same as the second frame of the 
input clip, because the clip will be compressed or 25 
expanded at this point in time and so a new output 
frame must be calculated by interpolation. For each 
frame in the output clip there will be a correspond- 
ing value along the x axis of the profile graph. 
Unless there happens to be 256 frames in the 30 
output clip, i.e. the same as the number of points 
on the x axis, the corresponding position of each 
frame on the x axis must be calculated. Thus, the 
first stage of the interpolating process consists of 
identifying the positron of an output frame with 36 
respect to the values of the x axis. This calculated 
position may be an Integer or it may be a fraction. 
If the position is an integer then the x value for the 
position is used to address the corresponding value 
in the look-up table and the corresponding y value 40 
is thus obtained. If however, the calculated position 
is a fraction, for example 3.5, the two locations, i.e. 
addresses 3 and 4, are addressed and two y val- 
ues are obtained. The corresponding y value for 
the fraction at x value is then obtained by inter- 4$ 
polating between the two y values output from the 
look-up tablet which gives y as a fractional number 
between 1 and 256. For a given value on the y axis 
there will be a corresponding frame position in the 
input video clip. The corresponding input frame so 
position is calculated from the y value to identify 
the frame or frames in the input clip which is or are 
to comprise an output frame. The calculated input 
frame position may be an integer in which case a 
single input frame forms the output frame, ir it may 55 
be a fraction In which case two input frames will 
contribute to the output frame. If the calculated 
frame position is a fraction then an interpolated 



output frame is derived from two stored input 
frames by linear interpolation of each pixel of the 
stored input frames using a weighting determined 
by the fractional component of the calculated input 
frame position. 

The process for generating new frames by 
interpolation will now be described by way of ex- 
ample with reference to Figure 3 of the accom- 
panying drawings, which shows a portion 27 of the 
profile 23 on an enlarged scale. Also shown in 
Figure 3 are axes representing output and input 
frame positions in relation the x and y axes respec- 
tively. In this example, a 300 frame output clip 
(corresponding to 10 seconds of NTSC video] is 
generated from a 200 frame input clip. 

The profile portion 27 covers x values from 41 
to 47, and corresponds to a range of output frames 
48 to 55 derived from input frames 100 to 106. For 
the purpose of explanation it shall now be assumed 
that the process has reached the stage where it will 
calculate output frame number 50. 

The X axis is divided into 256 integer values 
representing 300 output frames and therefore the 
position x{50) of output frame number 50 on the x 
axis may be expressed as a fraction of the total 
number of output frames: 
x(50) = 50/300 = 0.166 

The X values are actually stored at intervals of 
1/256 and therefore the x value xs{50) of output 
frame number 50 with respect to the look-up table 
address is: 

xs(50) = 256x0.166 = 42.66. 

The address 42.66 is not a true (i.e integer) 
address in the look-up table and the y value for this 
fractional x value is obtained by interpolating be- 
tween X =s 42 and x =43. From the look-up table, 
y = 129 when x = 42, and y = 131 when x = 43. 
Therefore, the y value for output frame 50, y(50), is 
given by: 

y(50) = 0.66 X 131 + (1 - 0.66) x 129 
= 86.46 + 43.86 
= 130.32 

The y axis is also divided into 256 values and 
represents an input clip having 200 frames. There- 
fore the input frame number 1(50). from which out- 
put frame number 50 will be derived, is given by: 
1(50) = 200 x (130.32/256) 
= 101.81 

Thus, output frame number 50 is derived from 
input frame numbers 101 and 102 and the frac- 
tional part of 1(50), i.e. 0.81. gives the weighting 
factor, i.e. an 81% contribution from each pixel in 
frame number 102 and a 19% contribution from 
each pixel frame number 101 on a pixel-by-pixe! 
basis. 

The output frame number 50 is thus generated 
by combining frames 101 and 102 of the input clip 
on a pixel-by-pixel basis for each luminance and 
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colour difference signal. 

Thus, each pixel P in the output frame number 
50 is calculated from: 

P(OUTPUT 50) = 0.81P(INPUT 102) + 0.1 9P- 
(INPUT 101) and the new frame data thus cal- 
culated Is stored In the disc store 10. The process 
is then repeated for output frame 51 and so on until 
the complete output video clip of 300 frames has 
been generated. 

All of the output frames are stored in the disc 
store 10 and can be read therefrom for display on 
the monitor 13 for review or they can be read for 
storage of the output clip off-line in a bulk record- 
ing device such as VTR 11. 

Having thus described the present invention by 
reference to a preferred embodiment It Is to be well 
understood that the embodiment in question is 
exemplary only and that modifications and vari- 
ations such as will occur to those possessed of 
appropriate knowledge and skills may be made 
without departure from the spirit and scope of the 
invention as set forth in the appended claims and 
equivalents thereof. 



Claims 

1. A video processing system, comprising a 
video storage device for storing an input sequence 
of video frames: defining means for defining an 
output sequence of video frames in relation to said 
Input sequence frames: and generating means for 
generating an output sequence of video frames 
from said input video frames in accordance with 
the defined relationship. 

2. A video processing system as claimed In 
claim 1, 

wherein said storage device comprises a disc store 
for storing said input video frames and for storing 
output video frames generated by said generating 
means. 

3. A video processing system as claimed in 
claim 1 . 

wherein said defining means is arranged to define 
a start frame and an end frame in said input video 
frames thereby defining an input video clip from 
which an output video clip is generated. 

4. A video processing system as claimed in 
claim 1 

further comprising a monitor for displaying either of 
or both said Input sequence of video frames and 
said output sequence of video frames. 

5. A video processing system as claimed in 
claim 1 , 

further comprising display means for displaying a 
graphical representation of the defined relationship. 

6. A video processing system as claimed In 
claim 5, 



wherein said display means is arranged to display 
substantially orthogonal axes representing input 
frame numbers and output frame numbers and to 
display a graph plotted against said axes repre- 
5 senting the defined relationship. 

7. A video processing system as claimed In 
claim 1, 

wherein said generating means is arranged to inter- 
polate a pixel-by-pixel basis on two input frames to • 
10 produce an output frame. 

8. A video processing system as claimed In 
claim 1, 

further comprising manually operable means for 
use in defining said relationship. 
75 9. A video processing system as claimed In 
claim 8, 

wherein said operable means comprises a stylus 
and touch tablet combination arranged to generate 
co-ordinate data relating to the position of the sty- 
20 lus on the touch tablet, which co-ordinate data Is 
interpreted as Identifying input and output video 
frames. 

10. A video processing system as claimed in 
claim 1 , 

25 further comprising storing means for storing dis- 
crete values defining said relationship and inter- 
polating means for interpolating between said dis- 
crete values to obtain related values between said 
stored values. 

30 11. A video processing system as claimed In 
claim 10, 

wherein said storing means comprises a random 
access memory device for storing said values as a 
look-up table. 

35 
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